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(54) Method and system for maintaining freeform ink annotations on changing views 



(57) The method and system of the invention main- 
tains the indication ol the togical relationship of freefonn 
di^l inkannotations to objects ort an eiectronc display 
oven as the view changes. The inventwn maps the tog- 
ical relatk)nsh'9> ^ freeform digital ink annotations 
to objects en the view based upon the teyout relation- 



ships of the freeform digital bik annoiatfons to the ob- 
jects in the view. If theviewchanges. the inventfon mafrj- 
tains the layout relattonshfp of the freeform di^ frtk 
annotatkxis to the objects in the view based upon the 
mapping of the freeform digital ink annolatkxts to the 
objects. 
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Description 

[pool] This application is directed to an electronic system tor displaying changing views. In paiticutar, the invention 
is (greeted to pemutting (reefonm digital ink annotation on elec^ic media and to maintantng the logical relationship 

5 between f reetorm digftal Ink annotatkms and objects in changing views of the media 

[0002] Freeform bik annotatkm of paper documents is a widespread practce because it aUs many aspects d doc- 
ument work, including the processes ol thinkng. categorizing and reviewing. In short, freefonn ink annotation has 
proven to be a powerful comprehensxsn tool Paper and paper-like documents (such as PDF. postscript, image files, 
etc.) teve a statk; layout and content. The static nature ol these documents makes annotation simple. 

to [0003] However, the kinds of documents thai people use are increasingly created and presented digitally on displays 
using cornputers. Such documents are often dynamic and, therefae, the accurate representatton ol the relatkxiship 
of the annotations to objects In the display has been impossible because of the changing display. For example, a 
display of a word processor document changes as the document is undergoing reviskins. In a similar mannec infor- 
mation visualization systems such as the hyperbolic browser cfisdosed in A Focus + Context Technkaue Based on 

IS HyperboBc Geometnr for Visualizing Large Hierarchies, by lumping et aL, m Proceedings of ACM SIGCHI Conference 
on Human Factors and Computing Systems, May 1 995. change the display of the document as the user navigates the 
tntomnation space. 

[0004] A freefonn digital ink whiteboard known as Tivoli is disck^&ed in Pen-Based Interaction Technwues For Or- 
ganizina Material on an Eledronfc Vtfh'rteboard. by Moran. et al., in UlST 'ST. ACM press. TivoG allows users to identify 
20 a connectky) between two areas in a displays where thm connection is maintai^^ 

display changes Ctrde nnarks which sumxind other ink identify areas, and freeform digital ink marks whk^h connect 
two area boundaries together fomr> a connection between those two areas. This connection persists as the areas move 
and Trvoli attempts to maintain the shape of the connecting mark. 

[00(^ To assist in navigatk)n of documents conventionat systems use a s^ type of bookmark. Examples of such 
^ navigatkm systems are disck>sed in Virtual Annotatkxi System, by Kamm. et aL. in Proceedings of the IEEE 1996 
Virtual Reafity Annual International Symposium, pp. 239-245 (1996) and in Wofkflets:3D Thumbnails for WavBndInQ 
in Virtual Environments, by Elvins, et al.. in video proceedings of UlST "Sff, pp. 21-30 (1997). These systems make it 
ackwaid for a user to freely interact with the navigation system because the user is required to specify an annotation 
command before commenting. 

^ [0006] Current systems attempt to manage f reetomn digital ink annotation on changing views or displays by at least 
one of three methods (the terms View* and 'display* are used interchangeably throughout this appricatbn). The first 
method, known as the static approach, treats ink annotations as associated with a single view When the presentatk>n 
or the content of the document changes, the annotations either disappear or reman m their original positiorts. A system 
known as 'MATE' Which Is disctosed in A Marking Based Interface for CoHaborative Wiritinft by Haidock et aL, in 

3S Proceedings of UlST *93 (1 993), addresses changing content by maintaining a static annotated version, a changed 
version without annotatkx) and bgical links between these two versions. 

[0007] A second method, known as the snapshot approach, treats the dynamic view as a series of static snapshots 
which can be indivkJually annotated. Examples of this approach include the Vfeng Freestyle disck>sed in Rapid. Inte- 
grated Design of a Multimedia Communication System, by Francik. in Human-Computer Interface Design, Morgan 

40 Kaufman PubBshers, Inc., (1996). 

[0008] The third method, known as the temporal associatkxi approach, separates the ink annotations from the view 
and links the two temporally, so thai marks made In a specific time are associated with the view at that time. Examples 
of this approach are used in Marquee which is disclosed in Marquee: A Tool for Real-Tinta Vtdeo Logging, by Weber 
et al., in Proceedings of CHI *94. ACM Press, (April 1994) and Dynomite whkdi is disclosed In Dvnomite: A Dvnamicallv 

4S Organized Ink and Audio Notebook, by Schilit et aL. In Proceedings ol CHI "97, ACM Press, (1 997). 

[0009] In conventional interfaces that accept freefonn digital ink annotation, the indkation of the k)gcal relationships 
between the annotations and the associated objects are not maintained il the view changes. For example, if a user 
annotates the text of a docuntent and, subsequently, the annotated text reftows across Ones, the annotatkxi does not 
fdtow the refiowed portion of the text. Therefore, while there may be an underlying bgk^l relatkmship to the annotated 

SO text, the layout of the new view does not indicate this \oQks\ relationship between the text and the annotatk>n. 

[0010] The systems described as related art above also do not maintain any indcation of togcal rslatkxishtps be- 
tween the annotatkxis and the associated objects if the view changes. Tivoli does not maintain the freefomt di^'tal ink 
annotations as the areas move around and the display changes. Rather. Trvoli maintains only one specific type of 
. annotation: a single stroke from the boundary of one area to another. This restrtetnn precludes the use of freeform 

ss digital ink annotations in changing displays to label moving objects or to akl navigatkxi. 

(0011] WhDe MATE maintains the logical relatk)nships between freefomndigital inkannotationsand the corresponding 

undertying text, it does not maintain the indicatkxts of these k)gical relatunships as the view changes. 

[0012] Freestyle does not altow the freeform digital ink annotattons tocarry over to new views of the same document 
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[001 3] DynGmita does not permit annotations directly on the view and the annotations are not rendered on the new 
views in aooordance with changes to the views. 

[0014] The method and system of the invention maintain the layout relationships between treefomi digital ink anno- 
tations and objects in changing views. As a user makes a f reeform digital ink annotation on a view and as the view 
5 changes, the method and system oi the invention adapt the marks in accordance with the changes. Thus, the k)gk»l 
relationship between the freeform digital ink annotatk)ns and the objects made in one view can be understood in future 
views based upon the layout retatkmshtps. 

[0015] The method and system also maintain the indicatkM) of the logical relatkxiships between freetorm digital ink 
annotations and objects in dynamic media such as vkteo and interactive virtual workSs. For example. H a user, who is 
to exploring an information space wdh a hyperbolk: browser, makes a freeform digital ink annotatkn on a node of the 
tree, when the user changes the display to bring another portkxi of the tree Into focus, the freef onn digital Ink annotation 
rotates and scales aksng with the tree tocontinue to indicate the k>gical relationship of the freef onn dgital ink annotatton 
to the node. 

[001 6] The methods and sy^ems of this inventkn maintain the indicatkxi of the togkal relatkxtships between freeform 
IB digital ink annotatkxis and objects in a view as the view of the annotated objects changes. This invention brings the 
experience of marking on paper to dynamk: media, changing wordi>rocessor documents, infoqnation visualizations, 
virtual workJs, digital video arxl vnxd processors. 

[0017] The system and method cA the present invention maintain the Mcation of the k)gical relationship, even if the 
-objects that are togically related to the freeform digital ink annotations are deleted, inserted, copied or moved or oth- 
envise altered. The method and system also maintain the indicatk^n of the togteal relatcnship if the layout presentatkm 
of the objects in the display change because the objects reftoMi scale, resize, rotate, stretch, distort or morph or if any 
other aspect d the layout presentatkm changes, regardless of whether these changes occur discretely or continuously 
[001 q Additionally; another exemplary embodiment of the systems and methods of the invention provkles naviga- 
tkanal markers that make freeform dtgital ink anrtotatkxi a powerful tool for navigating dynamu documents. A user is 
ss permitted to select a freeform digital ink annotatcn in an arbitrary mvi and the display is retumed to the state d was 
in when the selected freeform digital hk annotatkxi was made. 

[001 9] The preferred embodiments of this invention will be described h detail, with reference to the fotlowtng figures, 
wherein: 

30 Fig. 1 is a bkick diagram of an embodiment of the system for displaytng changing views of this inventnn; 

Fig. 2A ^xiws a display of an annotated hyperbolic browser in accordance with an embodiment of the present 
invention; 

Fig. 2B shows the annotated hypertoUc browser of Rg. 2A with the view changed and freeform digital ink anno- 

tatkns adjusted in accordance with the change in the view in accordance with an embodiment of the inventkm; 
OS Fig. 3A shows a display of an annotated document; 

Rg. 3B shows the annotated docun^ent of Fig. 3A after modriication of the document and shows the adjustments 

to the freeform digital ink annotations in accordance with an errdxxJiment of the present inventkxi; 

Rgs. 4A'4F show a flowchart outTtning a control routine for mapping f reefonm digital ink annotatk)ns to object in 

a display in accordance with an embodiment of the prevent invention; 
40 Figs. 5A-5B show a flowchart outlining a control routine for rendering freeform digital ink annotatksns on a chartged 

view in accordance with an embodiment of the present inventksn; and 

Rg 6 shows a few examples of cfifferent types of freeform digital ink annotations. These and other features and 
advantages of this inventxsn are described in or are apparent from the foltowing detailed dascriptbn of the preferred 
embodments. 

45 

[0020] Rg 1 shows one exentplary embodiment of the electronic system 1 0 of this tnventnn. The electronic system 
10 includes a processor 12 communk»ting with a memory 14 that stores the programs, documents and other data for 
practicing the inventkan. The processor 12 also communicates with a display 16. a keyboard 18, a mouse 20 and a 
pen 22 via inputAxitput device 24. The keyboard 18. mouse 20, pen 22 and any other interface devx:es (not shown) 

so are operated by a user to control the operatkxi of the electronic system 10. Control of the electronic system 10 is not 
Med to the interface devices shown and does not require any particular interface device. The display 16 may be on 
a stand-akme pen computer or a tablet connected by a tether to a conventional computer or any other type of display 
[0021] As shown m Fig. 1 , the system 10 is preferably implemented using a programmed general purpose computet 
However, the system 10 can also be implemented using a special purpose computer, a progranvned mcroprocessor 

ss or microcontroller and any necessary peripheral htegrated circuit elements, an ASIC or other integrated circuit, a 
hardwired etectronc or togic circuit such as a discrete element circuit, a progranunable logk: devk:e such as a PU), 
PIA FPGA or PAL^ or the like. In general, any device on whch a finite state machine capable of implementing the 
flow charts shown in Figs. 4A-5B can be used to implement the system 10. 
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[0022] Additionally, as shown in Fig. 1. the memoiy 14 is prefembty implemented using static or dynamic RAM. 
Howsvar. the memoiy 14 can also be implemented using a floppy disk and disk drive, a writable optical disk and disk 
drWe. a hard c^ive. a flash memory or the Uke. Additionally, it should be appreciated that the memory 14 can be either 
distinct portions of a single memoty or physfcally distinct memories. 
5 [0023] Further,itshoukJbeappreciatBdthatthellnk826,28,30.32.34amJ36connecth 

16, the keyboard 16, the mouse 20, the pen 22 and the inputA)utput device 24 to the processor 12 can be wired or 
wireless links to networks (not shown). These networks can be k)cal area nstworits, wide area networks, intranets, the 
Internet, or any other distrftMJted processing and storage networks. 

[002^ Fig. 2A shows a display of a view of a hyperbolic browser 40. The display 40 includes a f reef omn digftat ink 
10 annotation 42: Fig. 2B shows an alteniative view of the document shown in Rg. 2A. The treelonm digital ink annotatk)n 
42 has been scaled and moved so that the layout presentatkm of the f reefomi digital ink annotatkn in the view corre- 
sponds with the change of the view of the objects in the document 40 to maintain the indicatkm of the k)gical relationship 
between the freetomn digital ink annotatkxi 42 and the objects. 

[0025] 3A shows a first view 50 of a document 52 having f reef orm digital ink annotations 54, 56 and 56. Fig. 38 

IS shows a second view 60 of the document 52 after the document has been revised Fig. 3B shows that the Ireefom) 
digital ink annotatkins 54. 56 and 58 are ac^ed in accordance with the logical relattonsh^ to the objects in the 
document 52 in accordance with an embodiment of the invention. The content of the docunrtent 52 shown in Fig. 3A 
has been ecfited to insert the text "is so much fun to do* at 62 and to delete approodmatety two full Ones of text at 64. 
In response to these edits, the text that has been annotated at 54 now ficm across two lines and the *cross<Hjr 

20 f reeform digital ink annotatton 54 fksws along with the undertyfng text objects to continue to indkate the k)gKal reta- 
tk)nshlp between 'cross-out* freelorm di^ ink annotatkm and the underlying text objects. The invention also recog- 
nizes that the group of ink strokes at 50 is a margin note and the system slightly adjusts the vertical positnn of the 
otargin note 58 in the margin to continue to indicate the bgical relatkxishtp between the ntargin note 58 and the refkywed 
line. AdcfitionaOy. the freeform digital ink annotation 56 is ad!|usted to adapt to the one line being removed between the 

2S anchor points at 66 and 68 to continue to indicate its bgtcal relationship to the text. 

[0026] An embodiment of the method and system of the present hventkm involves at least a three-etep process. 
The first step is adding the freeform digital ink annotaton on objects In a first view The second step Involves mapping 
segments of the freefomn digital ink annotatkans to corresponding objects in the view The third step involves moving, 
scalfftg or otherwise transforming the freeform digital ink annotatkxi to maintain the hdication d the k>gical relationship 

30 between the freeform digital nk annotatkm and the corre^xmding objects as the display changes. 

[0027] Optkxtaliy, in an embodiment of the inventkxi the freeform digital ink annotations can be used as navigationat 
markers. For example, the user can select an annotation in a view and return to the annotated view that existed when 
the annotation was created. 

[0026] The freeform digital ink annotation of objects in changing views of an embodiment of the tnventon relies on 
3S an object model for the view. In the example of Figs. 2A and 2B, the objects are the nodes in the hyperbole browsec 
In the example d Rgs. 3A and 3B, the objects are the text characters. In a third example (not shown), the objects are 
the output from an object tracking system nmning on a vkfeo data stream 

[0029] An embodtment of the method and system of the invention maps the freeform digital hk annotations to objects 
in a view by placing anchors in the underlying document For example, a simple drdmg. hightighting or a crossing out 
^ mark is anchored to the conespondmg span of character pcsitnns that are covered by the freeform digital ink annotatkxt 
in the view. In the example shown in Figs. 3A and 36, a comment 56 in the margin is anchored to a posHkm in the 
document in the margin near aB of the con-esponding words or characters. 

[0O30] More complex freeform digital ink annotatkxis may have several anchors. For example, as shown in Figs. 3A 
and % oonsRter a freeform digital ink annotatkxi 56 that includes a circle 68 connected to an arrow 66. The drde 68 

4B is anchored to the span of words or characters that is selected by the circle and the tip of the arrow 66 is anchored to 
its end point while the fine connecting the circle 68 and the tip of the arrow 66 is not anchored. 
[0031] An even more complex freeform digital ink annotation may involve togical retationsh^ to rncwng objects in 
a virtual workl An exarnpleotf such an annotation involves the freeform digital inkannotatunofadoortoaroomina 
virtual workl The layout of the door changes in a view of the virtual wohd as the door is opened or closed. The method 

so and system of an embodiment of the inventkxi maintan the indicatkin cH the k)gk:al relationship between the f reefomi 
digital ink annolatbn and the door as the view of the door changes. Thus, the user sees the freefomi digital ink anno- 
tation rotating with the door. 

[0032] For the purposes of the folbwing discusston. the following definilkxis are pronded A positkyi is a two^iimen- 
skxial coordinate in a view. A stroke is a list of positkins in a view. An offset isa twodimenstonal offset from a positkxi 
ss in a view. A regkxi is a portkxi of a two dtmenskyial view that is represented, for example, by a bounding box or by a 
list of positkxYS. A character is a character in a text document Character positkxts between characters In the fkiw of 
text are maintained as the text changea A margin anchor is a character positksn and a boolean flag for left or right 
margins. A simple anchor is a character, a character positkxi or a margin anchor. A composite anchor is a pair of simple 
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anchors with offsets. An anchor is either a simple anchor or a composita anchor An anchored stroke b an anchor plus 
a list of offsets. 

[0033] An emtxidiment of the method and system d the Invention moves the anchor points H the underlying anchored 
objects move. Unanchored segments of the freefomi digital ink will stretch to fit I>etwe6n the anchored points. Some- 

s times adjacent anchor points move apart because the underlying objects move apart For example, if a text phrase 
reflows from a single fine across two lines (as shown in Figs. 3A and 3B). In such cases, the connection between the 
fomwrly accent anchored points can be left out to reduce clutter and any unanchored ink between those points can 
be removed Thus, as shown in Fig. 3B, the reftowed freelotm digital ink annotation is spot and does not show ink 
extending between the two segments, which are on two separate text lines. If anchor objects for a stroke are deleted, 

10 then the freelonm digital ink annotattons can be renrtapped to the nearest remain^g objects in the document If aO 
anchor objects for the f reeform digital ink annotatnn are deleted then the annotatkm can be deleted. 
[0034] The type of the anchors for the treeform digital ink annotatkx) depends upon the shape of the annotation and 
the kind and number of the corresponding objects. One method, detailed betowi for computing bik anchors is to select 
the nearest object for each segment in the ink. Attemativelyi the type of stroke, as detenmined from its shape, can tw 

IS used to determine the anchor objects. In another case, a freeform digital onk annotatkxi with nrujttipte anchors may 
cause unwanted distortkxis when the view changes. To avoid this problem, an embodiment of the present bivention 
groups segmente of the f reefonn (figital ink annotatkxi together to reduce the number of anchors. 
[0035] As briefly described earlier, an embodiment of the method and system of the present invention uses at least 
three processes. The first process receives freeform digital ink annotalkxis on the viewc the second process computes 

20 a \og^ mapping between objects in the view and the treetonn digital ink annotations, and the third process renders 
the freeform digital ink annotations on a view tl^ has changed. An optional fourth process selects freeform d^ital ink 
annotations to navigate to a previous stale. 

[0036] The first process tndudes sensing data pc^s from a pointing device, such as a mouse or a pen. rendering 
the ink strokes by connecting the points on the view, and grouping the ink strokes together to form a freeform digftal 
^ ink annotation. These ink strokes may be grouped, for example, by a simple time threshokl The second process 
receives the f reefonn digital ink annotatk)ns as input and maps between the objects and the f reekxm digital ink anno- 
talkxis. The third process updates the view when the document changes. This process uses the mapping to render 
the freeform digital ink anrxjtatkxi on the ot)jects as Uie view changes. 

[0O37] The procedures and the embodiments of the invention descrbedhersin assume that the objects do not change 
30 whde the user is inking. However, the WBtttod and system of the inventkxi is not Gmited to app&^ltons tn which the 
objects being anrtotated do not change. As an example, in the case of video, the video may be hatted while the user 
marks. Altematively. the video can oontcnue while the user marks and the state dt the objects at some point in t'sme 
can be used to interpret the objects and update the mapping between the objects and the Ireefonn digital ink annota- 
lkxis. 

3S [0O3q The fourth process takesa selectkxi ona viev^ determines if it corresponds toa f reefonn digital ink annotatkxi. 
and. (f it does, replaces the cunent view with the view which exsted when the freeform digital ink annotation was made. 
This process requires storing an additkxial mapping from the freef omi digital ink annotations to the state of the changing 
view 

[003S] The present invention recognizes that objects may be buitt from subK)bjects whkii may be spirt from each 
^ other One example is a text phrase that is built out of words. If a phrase has a f reefonn di^ ink annotation that 
circles the phrase and if that phrase refksws across two lines, the inventkxi recognizes this split and may either reftow 
the f reefonn digital ink annotatkxi by splitting it up into several segments, each of whk:h foOows the corresponding 
underlying words as they nwe. or the system may choose to draw fines between matching segments of the circle. 
Altematively, the freeform digital ink annotatkxi coukJ be replk:ated at a smaller scale on each Mndual word. 
45 [0040] A user can also copy a phrase to another paragraph, whk:hcoukl be considered a fonn of splitth^ 
entirety new and unrelated object without the circle annotatioa 

[0041] Conversely, objects can merge into larger entries. For exarr^e. il an unannotated word between two anno* 
tated words is deleted then the two annotated words "merge* to tonn an annotated phrase. The two annotatkxis join 
to form a single annotation of the merged phrase. 
so [0042] Objects may also disappear entirely. For example, words in a text editor can be deleted. In this case, the 
freefomn digital ink annotatkxis associated with the object can disappear also, or they can remain at the kxatkxi be- 
tween the surrounding objects. 

[0043] Objects can also change. For example, a user in a text editor can select an annotated word and replace it 
with a new word. The new word couM be consktered a a new verskxi of the original object The annotatkxi on the 
55 original word may then be associated with the new object. In this case, the annotatkxi may alter its shape to conform 
to thanewword. for example, by stretching or shrinking. Altematively, the annotatkxi could be removed from the display. 
[0044] An embodiment of the method and system of this inventkxi may group ink strokes on text to form a single 
freeform digital ink annotation. ^ way of non-limiling examples, the annotations can be categorized as circles, under- 
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lines/cross^Mits, {nsertions, margin notes, and connectors. Annotations that are circles are identffied tiy ink stroke 
cirdes that contain text and these inkstrokecirdesarernapped to the text obj^^ 

cross-out annotatksns are identified by elongated marks which are either directfy on the text or underneath the text 
These annotatk)ns are mapped to the text objects that they are on or underneath. An annotation may be identified as 
an insertion 3 the annotations dips intoa line of text An insertion annotatkxi maps to the position where the insertion 
wajk appears in the text, for example, between two charaders. A margin note is an annotatkxi which is tocated in a 
margin. The margin note annotations are mapped to the text that is acQacent to the vertical and/br horizontal positron 
of the annotations. Connector annotatksns are kJentified as oonnedors by arrows that may appear at the end of the 
annotatton or by lassos (i e., a cirde plus a Bne and a single stroke). A connector annotation is spirt into a drde and a 
oonnedor portbn. Connector annotations are mapped to two objects Independently. The first object is the objsd that 
occurs at the start of an annotation and the second objed occurs near the end d the anrxM^^ 
be posHkms in the text, or they may be other annotattons. As an exanrtple. one end of the Ihe portkxt of a lasso is 
mapped to the cirde annotatkxi whch. in turn, maps to the text. Another example is a line from the text to a margin 
annotatkxi. One end of the line is mapped to the text while the other end is ma|^ to the margin annotatkxi. 
[00451 Rg. 6 shows several exemplary types of f reefomi digital ink annotations. Fig. 6 shows a display of a document 
400 induding circle (or endosure) annotations 402 and 404. an underline annotation 406. cross^ annotations 408 
and 41 0. insertkxi annotatkxis 41 2 and 41 4. a margin annotatkxi 41 6. a Kne to margin annotation 41 8 (with assodated 
margin note 420) and an arrow connector 422. It shouM be understood that the types of free! orm digital btk annotatkxis 
shown in conjundkxi with the descriptkxis of various exemplary embodiments herein are intended to be Ohjstrative 
and not limiting to any particular embodiment/type or grammar of annotatkxi. 

[0046] TTie method and system of this inventk>n is based upon the concept that anchors are detemiined for f reetorm 
digital ink annotations. Anchors are offsets from objeds. Freefomn digitat ink annotatkxis are mapped to the objed 
structure of the view underlying the f reefomi dgital ink annotatons. While it b to be understood Uial the mapping may 
be to hierarchical objed stnidures such as characters, phrases and paiagraphs. for the sake of simplk% the fdtowing 
descr^txx) describes a mapping of a single level; between f reetonn digital ink annotatkxis and charaders or positkxis 
between charaders. A text editor maintains these charaders and positkxis as the text changes. For charaders that 
have been deleted, the text editor marks the charaders and their corresponding charader positkxis as deleted, but 
the text editor stin maintains a correspondence between the charader positkxis and some kxratkxi between remaining 
charader& 

[0047] An entfxxtiment of the method and system of the inventkxi may rely upon a text editor that stores the sequence 
and presentatkxi of charaders. The embodiment can determine if a character position, a character or a range <^ char- 
aders has been deleted; can acquire the positkxi in the view that corresponds to a character positkxi; acquire the 
region in the view that corresponds to a charader; can obtam left and right mar^n positkxis that correspond to a 
charader position; can acquire a range of charader positkxis that correspond to a regksn in a view; and can split a 
f reeform digital mk annotatkm Into a set of substrokes or segments, such that each substroke corresponds to a single 
charader or to a positkxi between charaders. 

[PO40| One embodiment of the method and system of the invention may also use an annotation database that stores 

anchored strokes. The annotation database may be accessed to add an anchored stroke to the database, to delete 

an anchored stroke from the database or to acquire a list of all of the anchored strokes. 

[0049] Rgs. 4A through 4F show a flow chart outlining a control routine for mapping freeform digital ink annotations 

to objeds in an underty ing view in accordance with an embodiment of the inventnn. The flowchart parallels the following 

pseudocode. 
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First, classify the group of strokes as drcles, underlines/oos&^uts, inseitioiis, margin 
notes^ or coimectors. Notice that this may require splitting some of the strokes* and 
may result in several groiqis of strokes. Then, 

Sort the new groups of strokes so that connectors q>pear after any groups they 
connect to 

For each new group of Strokes 

If the group is a circle or an underline or a cross-out, then 

Split the grotq) of strokes into a list of strokes and characters 
For each stroke and charuter pair in the list 

Create a new simple anchor corresponding to the character 
Get the region corresponding to the character 
Let anchor position be the center of the region 
Create a new anchored stroke based on the sinqple anchor 
For each position in the stroke. 

Let offiet be position minus andior position 
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Append ofiset to the anchored stroke*^ list of ofbets 
End for 

Add anchored stroke to the annotation database 
End for 

Else if the group is an ink insertion into the text, then 

Let characttt position be the character position where the ink dips into the 
text 

Get the position corresponding to the character position 
Let anchor position be that position 
For each stroke in the groq> 

Create a new anch(»ed stroke based on the character position 
For each position In die stroke 

Let oflset be position minus anchor position 
Append offiet to the anchored stroke's list of of&ets 
End for 

Add anchored stroke to the annotation database 
Endfor 

Ese if the group is a margin annotation, then 

Let character position be the first character position corresponding to the 
vertical extent of the annotatioa 

Get d» left and lig^ margin positions coneq>onding to character position 
If the group is mostly to the left of the left margin position, then 
Let margin anchor be ( character position, flag=lcft) 
Let anchor position be the left margin position 

Else 

Let niargin anchor be { character position, flag^right } 
Let anchor position be the right margin posidon 

End if 

For each stroke in Regroup, 

Create a new anchored stroke based on the margin anchor 
For each position in die stroke 

Let of&et be posidon minus anchor position 
Append oflset to die anchored stroke's list of oftets 
Endfor 

Add anchored stroke to the annotation database 
Endfor 

Else if the groiq> is a connector, then 

For each of starting and ending position on connector 
If position is near an annotation 

Let simple anchor be the anchor for that annotation 
Let anchor position be the current position of the simple 
anchor 

Let offset be position minus anchor position 
Else (position is near an object) 

Let simple anchor be the anchor for that object 
. Let anchor position be the current position of the object 
Let of&et be position minus andior position 

End if 

Endfor 
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Let composite anchor be the anchors and of&ets conesponding to the 
starting and ending positions 
5 For each stroke in the group 

Create a new anchored stroke based on the composite anchor 
For each position in the stroke 

Let of&et be position minus starting position 
10 Divide offset by the distance between thestarting and 

ending positions 

Rotate of&et around (he origin by-angle (endmg position 
minus starting position) 

,5 ^ypend of&c^ to the anchored stroke's list of offiets 

Qidfor 

Add andiored stroke to the annotation database 
End for 

20 End if 

End for 

[0050] The control routine shown in Rgs. 4A-4F starts at step SlOO and continues to step S102. In step SI 02. the 
control rotitine groups the ink strokes into annotations such as circles, underfines^^ross-outs, insertions, marginalia or 

^ connectors, tt is to be appreciated that this may require splitting some strokes and may result m several groups of 
strokes. The control routbie then continues to step SI 04 where ttta annotations are sorted so that connectors appear 
after any group to which they are connected. The control routhe then continues to step SI 06 where ttie control routon e 
initiates a tocp that iterallvely processes alt of these annotations. The control routine determines if the current annotation 
is a ctrcis, an underthe or a cross-out. If in step SI 06, the control routine determines that the current annotation is a 

30 drde, an underline or a cross-out then the control routine jumps to step S1 16. If, on the other hand, in step SI 06 the 
control routine determines that the current annotation does not include a circle, an underline or a crosscut then the 
control routine continues to step SI 08. In step SI 03, the control routine determines 3 the cun'ent aruiotatksn is an ink 
Insertion. If, In step SI 08, the control routine determines that the current annotation is an ink insertion then the control 
routine jumps to step SI 38. If, on the other hand, the control routone determines that the current annotation is not an 

3S ink insert nn, then the control routine continues to step S11 0. In step S1 10, the control routine determines ? the if the 
current annotation is a margin annotation. If, in step S110, the control routine detennines mat the current annotation 
is a margin annotatnn then the control routine jumps to step S158. If, on the other hand, in step S 110, the control 
routine determines that the current annotatkm is not a margin annotatkxi then the control routine jumps to step S186. 
[0051] After completing one <A the processes of Figs. 4&4F. which will be explained in detail bekw, the control 

<0 routine continues to step Si 12 where the control routine detemtines if more annotations need to be processed If in 
step S1 1 2, the control routine determines that more annotatkx^ need to be processed, then the control routine returns 
to step S106. It. on the other hand, the control routine determhes that no more annotattons need to be processed, 
then the control routine continues to step S1 1 4 where control is returned to the environment m whbh the control routine 
of Rg3.4A4r operate. 

45 [0052] At step S1 1 6, the control routine spirts the annotatnn into a list of substrokes each with a corresponding single 
character. Thus, the control routine spirts the annotatkyi into substrokeA^haracter pairs. The control routine then con- 
tinues to step S1 18, where the control routine initiates a bop to iterativety process each substroke^character pair in 
the current annotatkxt. In step S118, the control routine creates a new sirhple anchor that corresponds to the cun^t 
character. The control routine then continues to step SI 20 where the control routine acquires the regnn in the view 

so that corresponds to the current character. The control routine then continues to step SI 22 where the control routine 
sets an anchor positnn to the center of the region in the view The control routine then continues to step SI 24 where 
the control routine creates a new anchored stroke that is based upon the simple anchor and continues to step SI 26. 
In step S126. the control routine sets an offset to the position from the substroke minus the anchor positkm and con- 
tinues to step SI 28. In step SI 28. the control routine appends the offset to the anchored stroke's list of offsets and 

ss continues to step SI 30. In step SI 30, the control routns determines whether more positkyts exists m the current 
substroke. If the control routine determines in step SI 30 that more positions exists in the current substroke, then the 
control routine retums to step SI 26. if. on the other hand, the control routine determines that more posilkxis do not 
exist in the current substroke then the control routine continues to step Si 32. In step SI 32, the control routine adds 
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the anchored eiroke to the annotation database, and the control routine then continues to step S134. In step S134. 
the control routine determines if more substrolce^bhaiacter pairs exist in the current annotation* II. in step S134. the 
control routine determines that more substrolce^character pairs exist in the current annotation then the control routine 
returns to step S118. if. on the other hand, the control routine determines, in step S134. that no more sut>strol(a^ 
s character pairs exist in the current annotation then the control routine continues to step SI 36 where control returns to 
step S112. 

[0053] Atstepsisathecontrolrouttnesetsacun^entcharacterpositiontothecharacterposnionwhere 

dips into the text. The control routine than continues to step SI 40 where the control routine acquires the position 

corresponding to the current character posftion. The control routine then continues to step S142 where an anchor 

10 position is set to the current character positen. The control routine then continues to step SI 44 where the control 
routine initiates a loop to process all strokes in the annotation. In step SI 44. the control routine creates a new anchored 
stroke based upon the current character position as an anchor and the control routine oontinues to step Si 46. In step 
S146, the control routine sets an offset to the current character position from the stroke mini^ the anchor positnn and 
continues to step S1 48. In step S148, the control routine appends the offset to the anchored stroke's fist ol offsets and 

IS continues to step SI 50. In step SI 50. the control routine detennines whether more positkvis exists in the current stroke. 
If. in step S1S0, the control routine determines that ntore positions exist in the cun'ent stroke then the control routine 
returns to ^ep S146. If, on the other hand, in step SI 50, the control routine detenmhes that no more posrtkxis exist 
in thacurrent stroke then the control routine continues to step S152. In step SI 52. the control rout^e adds the anchored 
stroke to the annotation database and the control routine continues to step S154. tn step S154. the control routine 

so detennines whether more strokes exist in the currem annotatkia 11. in step S154, the control routine delemiines that 
mae strokes exist in the current annotalkm then the control routine returns to step S144. If, on the other hand, the 
control routine determines, in step SI 54. that no more strokes exists in the current annotation then the control routine 
continues to step 8112. 

[0054| At step S158. the control routine sets a character position to the first character posrtkxi corresponding to the 
^ vertcal extent of the current annotatioa The control routine then continues to step SI 60 where the text editor acquires 

the left and right margin posHnns that correspond to the character positk)n and continues to step SI SZ In step S162. 

the control routine determines whether the annotation is mostly to the left of the left margin. If, in step S162, the control 

routine determines that the annotatkm is mostty to the left of the left margh. then the control routine continues to step 

SI 64. If, on the other hand, the control routine determines that the annotatkxt is not mostly left of the left mar^ then 
00 thecontrol routine jumps to step S1 68. Instep S164. the control routine sets the margin anchor to LEFT and the control 

routine continues to step S166. In step SI 66, the control routine sets an anchor positkyi to the left margin position and 

continues to step S172. 

[0055] In step S166. the control routina sets a margin anchor to HGHT and continues to step S170. In step S170. 
the control routine sets an anchor positfao to the right margin position and continues to step S172. In step S172, the 

3S control routine initiates a kx3p to process all strokes on the current annotation. In step SI 72, the conud routine creates 
a new anchored stroke based upon the margin anchor and continues to step SI 74. In step S174. the control routine 
sets an offset to the position from the current stroke minus the anchor position and continues to step SI 76. tn step 
S176, the control routine appends the offset to the anchored stroke's list of offsets and continues to step S178. In step 
S178. the control routine determines whether more positkms exist in the current stroke. If. In step S178. the control 

<o routine determines that more positions exist in the current stroke then the control routine retums to steps S174. If, on 
the other hand, in step S178, the control routine determines thai no more positions exists in the current stroke then 
the control routine continues to steps Si 80. In step SI 80, the control routine adds the anchored stroke to the annotation 
database and continues to step S182. In step S182. the control routine determines whether more strokes exists in the 
current annotatioa If, in step S182, the control routine determines that more strokes exists in the cun^ent annotatkxt, 

^ then the control routine retums to step S172. If, on the other hand, the control routine detennines, in step S182, that 
no more strokes exists in the current annotation then the control routine continues to step S156 where the control 
routine returns to step S1 12. 

[00561 At step S186. the control routine initiates a kxip to process all starting or ending positions in the currant 
annotatnn. In step $186. the control routina determines whether the current of the starting and endffig positkms are 

so near another annotation or an object If in step SI 86, the control routine detemitnes that the current positkxi is near 
another annotation then the controller continues to step S188. It. on the other hand, the control routine determines, in 
step SI 86. that the position Is not near another annotation then the control routine continues to step SI 92. In step 
S188, the control routine sets a simple anchor as the anchor for the current annotation and continues to step S190. In 
step S 190. the control routine sets an anchor positkm as the current posittoo of the simple anchor and continues to 

ss step SI 96. In step SI 92, the control routffie lets a simple anchor be the anchor lor an obje(^ which is near the current 
starting or ending positkxi and the control routine continues to step S194. In step S194. the control routine sets an 
anchor position to the current posHkxi of ttie object and the control routine continues to step Si 96. In step S195, the 
control routine sets an offset to the currant positkxi minus the anchor position and continues to step SI 98. In step 
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S138. the control routine determines wttether more starting or ending positkms exist in the current annotation, if. in 
8^ep S19&. the control routine detdrmines that more starting or ending positions exist in the current annotation then 
the control routne returns to step Sl86. If. on the other hand, the control routine determines, in step S198. that no 
more starting or ending positions exist in the current annotation thsn the control routne continues to step S200. 
[0057] In step S200. the control routine Initiates a loop to process all strokes in the current annotation. In step S200, 
the control routine creates an anchored stroke having a cornposrte anchor based upon the anchors corresponding to 
the starting and ending positions and continues to step S202. In step SSX)2, the control routine sets an offset to the 
current positnn in the current stroke minus the starting posftion and continues to st^ S204. In step S204. the control 
routine dhndes the offset by the distance between the starting and ending positions and continues to step S206. In 
step S206. the control routine rotates the offset around an origin by an angle equal to the ending posilkm minus the 
starting position and continues to step S20B. In step S208. the control routine appends the o8set to the anchored 
stroke's &st of offsets and continues to step S210. In step S210, the control routine detemiines whether more positksns 
exists in the current strok& It, in step S210. the control routine detennnes that more poeittons exists in the current 
stroke then the control routine returns to step S202. If, on the other hand, the control routine determines, in step S210. 
that no more positions exists in the current stroke then the oontiol routine continues to step S21Z In step S212, the 
control routine adds the composite anchor to the annotatnn database and continues to step S214. In step S214, the 
control routine detemunes if more strokesexists in thecurrent annotatkn If, in step S214, the control routffiedetermines 
that more strokes exists in the cunent annotation then the control routine returns to step S200. If, on the other hand, 
the control routine determines that no more strokes exists in the current annotation, then the control routine continues 
to step S216 where the control routine retums to step S112. 

[OOSq Figs. 5A-5B shows a control routine for rendering freeform digital hk annotatkxts on a changed view The 
flow chart parallels the folk>wnDg pseudocode. 

Using the annotation database, get tbe list ofall anchored strokes 
For each anchored stroke, 

Extract the anchor and the list of ofibets 
Let stroke he a new stroke 
If anchor is a simple anchor 

Dctomine if the character or character position underlying the anchor 
has been deleted 
If not, then 

Let anchor position be the position of that anchor 



11 



EP 1016 983 A2 



For each offset, 

Let new position be ofket + anchor position 

Add new position to the stroke 
End for 

End if 

Else 

Determine if the character or character positions underlying the 1 and 

anchors has been deleted 
If both have been deleted* then 
do nothing 

Else 

stalling position be the position of the l"* anchor, plus the 
I'offict 

Let ending position be the position of the 2"* anchor, plus die 

2"*ofiBet 

Foreachoffi^ 

Let new ofto be offset rotated byangle (ending 

position minus starting position) 

Multiply new ofiset by the distance between the starting 

and ending portions 

Let new position be new ofiset plus starting position 
Add new position to the stroke 
End for 

End if 

End if 

Render stroke on the view 
End for 

[0059] The control routine starts at step S300 and continues to step S302. In step S302, the control rotitine acquires 
a list of aD anchored strokes from the annotation database and continues to step S304. In step S304, the control routine 
extracts the anchors and the list of offsets from the current anchored strc^e and continues to step S306. In step S306. 
the control routine lets the stroke be a new stroke and continues to step S308. In step S308, the control routine deter- 
mines whether the anchor is a sinrtpta anchor. If, n step S308, the control routine determines that the anchor is a simple 
anchor then the control routine continues to step S310. If. on the other hand, the control routine detemiines. In step 
8308, that the anchor is not a simple anchor then the control routine jumps to step S326. In step S31 0, the control 
routine determines if the character or character positksn has been deleted, tf. in step S310. the control routine deter* 
mines that the character or character position has been deleted then the control routine jumps to step S320. If, on the 
other hand, the control routine determines that the character a the character position has not been deleted then the 
control routine then continues to step S312. tn step S312, the control routine sets me anchor position to the position 
of that anchor for the offset and continues to step S314. In step S314, the control routine lets the new position be the 
offset from the stroke plus the anchor position and continues to step S31 6. In step S31 6, the control routine adds the 
new posttton to the stroke and continues to step S318. In step 8318. the control routine determines whether there are 
more offsets for the anchor, ff, in step S316. \he control routine determines that there are more offsets then the control 
routine retums to step S314. II. on the other hand, the control routine detemiines that there are no more offsets then 
the control routine jumps to step S320. In step S320. the control routine renders the stroke on the view and continues 
tostepS322Jn step S322,Uiecontrdromine determines if there are rnore anchored strokes in theannota^^ 
If. in step S322. the control routine detemnines that there are more anchored strokes in the annotatkxi database then 
the control routine retums to step S304. If, on the other hand, the control routine, detenmnes in step S322, that there 
are no more anchored strokes in the annotatkin database then ttie control routine continues to step S324 where the 
control routine retums control to the environment tn whk:h the control routbie operates. 
[0060] In step S326. the control routine determines rl the chamcter or character positkxi underlying the first and 
second anchors been deleted. If. in step S326. the control routine determines that both of these character or character 
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posttbns have been deleted then the control routifte returns to step S320. If. on the other hartd, the control routine 
determines, in step 832$, that the character or character position underlining the Hr^ and second anchors have not 
t>een deleted then the control routine continues to step 8328. In step 8328, the control routine sets the starting position 
to the position of the first anchor plus the first offset and continues to step 8330. In step S330. the control routine sets 
the ending posftlon to the position d the second anchor plus the second offset and continues to step 8332. In step 
8332. the control routine sets the new offset to the offset from the anchored stroke rotated by an angle equal to ending 
position minus starling position and the control routine continues to step 8334. In step 8334, the control routine mul- 
tiplies the new offset by the distance between the starting and ending positions and continues to step 8336. In step 
8338. the control routine sets the new position to the new offset plus the stalling position and continues to step 8338. 
In step 8338* the control routine adds the new position to the stroke and continues to step 8340. In step 8340, the 
control routine determines whether more offsets exist If, in step S340, the control routine detemiines that more offsets 
exist then the control routine returns to step 8332. If, on the other hand, the control routine detennines. in step 8340. 
that no more offsets exists then the control routine returns to step 8320. 

[00611 Freetomi digital ink annotatkxi is intended to mean a visible ink stroke of arbQraiy shape made with a pea a 
mouse or other pointing device that persists over time. A pen based nput device is intended to include a pen tablet, 
a pen display, a pen display tablet, a pen computer and^ a touch screea The systems and methods of this inventk)n 
nr\ay infer some meaning from the shapes or the morphotogkal chaiacteristfcs of ink strokes but do not renwve the ink 
stroke once they have been reoo^ized 

(00^ White the detailed description refers generally to electronic systems, it b ontended to be understood that the 
method and system of the present Mention may be applied to any electronic book and document reader that displays 
changing views. Electrons book and document recKlers include any etectronk: handheld devbe that can be accessed 
with a pointing devk» such as a pen and that accepts freeform digital ink as input Electronic book and document 
readers may also include other adcfitkmal features without limitatkxi. 

[0063] It is intended to be understood that the tenn nexT is intended to include text, drgital ink. audk), video or any 
other content of adocument indudng the document's stnicture. An object is intended t 

such as, by way of non-limiting example, a character, a word, a phrase, a sentence, a paragraph, a line, a sectton. a 
chapter, a portbn of an image thai is kjentifiable by a con^uter vision system in a single image or in a stream of images 
asinavkleo,et& 

[0064] It should be understood that a change in the structure of a document Is intended to include inserting, deleting, 
copying and movong of objects or the like. It is also intended to be understood that a change in the layout stnidure of 
a document includes reftowing of objects, resizing of objects, rotatkxi of objects, stretching of objects, distortkm of 
objects, morphing objects or the like. 

[D065| While this inventkm has been descn'bed in conjunctkyi with the specific embodiments outlined above, it is 
evktent that many attematives, modificatkjns and variatwns are apparent to those skiited in the art. Accordinglyi the 
preferred embodiments of the inventkm as set forth above are intended to be Dtustrated and not limiting. \Mous chang- 
es may be made without departing from the spirit and scope of the Invention. 



Claims 

1. A method for maintafrting an indication of a logicat relatkxishq) d at least one freeform digital ink annotatkxi to at 
least one object in a first view as the first view changes, the indlcatkm inctudtng a layout relationship of the at least 
one treefomi digital ink annotatnn to the at least one object, the method comprising: 

mapping the k)gk»l retatky^^ of the at least one freeform digital ink annotation to the at least one object 
based upon the layout relationship of the at least one f reefonm digital ink annotatk}n and the at least one object 
in the first view; 
generating a second view; 

rendering the at least one freefonn digital ink annotatkxi in response to the generation of the second vievt^ the 
layout relationship of the at least one freeform digital ink annotation and the at least one object in the second 
view based upon the mapped k)gical relationship of the at least one freefonn digital ink annotatkxi to the at 
least one object in the first view. 

2. The method of claim 1 , wherein the second view is generated in response toat least one of a change in the structure 
of a document and a change in the layout of the document 

3. The method of claims 1 or 2, wherein mapping the k)gu;al retatkxiship comprises: 
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splitting the at least one freeform digital Ink annotation into a plurality d sut)strokes, each substroka corre- 
sponding to a poition of the annotation that Is in proximity loan object; and 

creating an anchor fer each of the pturaifty of 8ut>strokes based upon the layout retatlonshtp of each of the 
plurality of eut^strokes to at least one object in the first view 

4. Themethodof claims, wherein: 

rendering the at least one freeform digital ink annotation comprises rendering the at least one freeform digital 
ink annotation in response to a separation of the plurality of objects in the second view; and 
splitting the at least one freerorm digital ink annotatkxi further comprises splitting the at least one free form 
digital ink annotation into at least two freefonn digital ink annotations, each freeform digital ink annotatksn 
corresponding to at least one of the plurality of objects. 

5. The method of any of claims M, wherein: 

the at least one object comprises at least two objects; and 

genemting the second view comprises generating the second view in response to at least one of merging at 
least two of the at least two objects and replacing at least one object of the least two objects with at least 
one new object 

& The method of any of daims 1-5. further comprising mapping the togical relattonship based on at least one of a 
shape of the at least one freeform digital ink annotatksn, a proodmily of the at least one freeform digital ink anrtotation 
to the object, and a kxation of the at least one freefonn digital ink annotatnn in the first view 

7. A system for maintaining an indication of a lo^cal relationshp of at least one freeform digital ink annotation to at 
least one object in a view as the view changes, the indicatnn including a layout relationshp of the at least one 
freeform digital ink annotatnn to the at least one object, the system comprising: 

a display that d^ys the layout relatkinship of the at least one freeform digital ink annotation to the at least 
one object; and 

a processor that maps the at least one freeform drgitat ink annotation to the at least one object based upon 
the layout retatkmsh^ of the at least one freef omn digital ink annotalnn to the at least one object tn a first view 
the display [being] responsive to a change in the first view to generate a second view and to render the at 
least one freeform digital ink annotatkan in the second view; the layout relatkinship of the at least one freeform 
digital ink annotation and the at least one object in the second view based on the mapped k>gical relationsh^ 
ol the at least one freefonn digital ink annotation to the at least one object in the first view 

a The system of claim 7, wherein the processomiaps the at least one freefonm digital ink annotatkxi by splitting the 
at least one freeform digital ink annotatkm into a plurality of substrokes, each substroka corresponding to the 
portion of the freeform di^ ink annotatkyi that is in proximity to an object, and by creating an anchor tor each 
of the plurality of substrokes based upon the layout retationshq) of each of the plumlity of substrokes to at least 
one object in the first view 

a The system of claims 7 or 8, wherein the processor generates the second view in response to at least one of the 
at least one object being split into at least two objects and the at least two objects being separated, the processor 
spllttEng the at least one freeform digital ink annotatkm into at least two freefonn digital ink annotations, each 
freefonm digital mk annotatkm corresponding to at least one of the at least two ot)ject3. 

10. The system of any of claims 7-9, wherein: 

the at least one object comprises at least two objects; and 

the processor generates the second view in response to at least one of at least two of the at least two objects 
being merged together and at least one of the at least two objects being replaced with at least one new object. 

11. The system of any of claims 7-10, wherein the processor maps the at least one freeform digital ink annotation 
based on at least one of a shape of the at least one freefonn digital ink annotatton, a proximity of the at least one 
freeform digital ink annotatnn to the at laast one object and a kxalksn of the at least one freefomi digital ink 
annotatton in the first view. 
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12. The system of any d claims 7-1 1 , wherein: 

the at least one (reefomi digital ink annotation includes at least two freefomn digital ink annotations and the at 
least one object includes at least two objects, each object corresponding to one of the at least two freetorm 
s digital ink annotattons; and 

the processor merges at least two of the at least two freefonn digital ink annotatkxis into a single freefonn 
digital ink annotatkm in response to at least two of the at least two objects being n^erged. 
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